Comparing Information-Theoretic Attribute Selection Measures: A Statistical Approach

In a previous paper (Lopez de Mantaras, 1991), we introduced a new information theoretic attribute selection method for decision tree induction. This method consists in computing for each node, a distance between the partition generated by the values of each candidate attribute in the node and the correct partition of the subset of training examples in this node. The chosen attribute is that whose corresponding partition is the closest to the correct partition (i.e. the partition that perfectly classifies the training data). In the paper we had also formally proved that such distance is not biased towards attributes with a large number of values in the sense specified by Quinlan in (Quinlan, 1986) and we had also some initial experimental evidence that the predictive accuracy of the induced trees was not significantly different from that obtained with the most widely used information theoretic attribute selection measures, that is, Quinlan ́s Gain and Quinlan ́s Gain Ratio. However, it seemed that the distance induced smaller trees especially when the attributes had different number of values. In that paper we could not confirm that the differences were statistically significant due to the small number of experiments we had performed. Now in this paper we report experimental results that allow us to confirm that the distance induces trees whose size, without losing accuracy, is not significantly different from those obtained using Quinlan's Gain but smaller than those obtained with Quinlan's Gain Ratio. These experimental results are supported by a statistical analysis performed using two statistical hypothesis tests : the sign test and the signed rank test.